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(54) Lane marker recognizing apparatus 

(57) A lane recognition apparatus for recognizing 
lane markers on a road surface based on a pair of stereo 
images of a scene in front of an own vehicle obtained 
from a stereoscopic camera, includes a stereo image 
processing means for calculating distance information 
of an image based on the pair of the images, a lane de- 
tecting means for detecting a lane marker of a lane in 



the Image based on brightness information of the image 
and distance information calculated by the stereo image 
processing means and a lane position recognition 
means for recognizing a position of the lane marker in 
real space based on a position of the lane marker de- 
tected by the lane detecting means and distance infor- 
mation calculated by the stereo image processing 
means. 
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Description 

BACKGROUND OF THE INVENTION 
1. Field of the invention 

[0001] The present invention relates to an apparatus 
for recognizing road configurations ahead of an own ve- 
hicle by identifying lane markers drawn on road surfac- 



2. Background art 

[0002] Recently, people have shown an interest in an 
apparatus for monitoring vehicle surroundings (herein- 15 
after, referred to as vehicle surroundings monitoring ap- 
paratus) for recognizing road configurations in front of 
an own vehicle using a preview sensor such as nrw>noc- 
ular camera, stereoscopic camera and the like. In order 
to recognize the road configuration, it is important to de- 20 
tect lane markers (white markers, no passing lines and 
the like) drawn on road surfaces. In case where a mo- 
nocular camera is used as a preview sensor, the detec- 
tion of lane markers is performed only based on infor- 
mation of two-dimensional picture image, that is, bright- 25 
ness of image. However, the picture image obtained 
from the monocular camera does not provide distance 
information to an object. Accordingly, there is a possi- 
bility that objects having a brightness characteristic sim- 
ilar to lane markers, such as pillars of preceding vehi- 30 
cles. guardrails and the like, are recognized as lane 
markers. 

[0003] On the other hand, in case where a stereo- 
scopic camera is used in place of a monocular camera, 
it is possible to obtain distance information to objects 35 
from a pair of picture images. This is absolutely advan- 
tageous over the monocular camera. Japanese Patent 
Application Laid-open No. Toku-Kai-Hei 5-265547 dis- 
closes a technology in which a distance image (distribu- 
tion characteristic of distance in picture images) is cal- 40 
cu^ated based on a parallax in stereo images and ob- 
jects (preceding vehicles, road surfaces and the like) are 
recognized only from the distance image. 
[0004] In case where the lane markers are recognized 
only based on the distance image obtained from the 45 
stereo images, there is a possibility that objects having 
the configuration similar to lane markers and the differ- 
ent brightness characteristic are recognized as lane 
mari<ers, because only rough configurations of objects 
are obtained from the distance distribution built in the so 
distance image. Such objects, for example, are traces 
of tire slips, shades, patches and the like. 

SUMMARY OF THE INVENTION 

55 

[0005] It is an object of the present invention to pro- 
vide a lane recognition apparatus capable of recogniz- 
ing the position of lane markers in the real space using 



both of the distance information obtained from a pair of 
picture images and the brightness information in the pic- 
lure images. Further, it is another object of the present 
invention to enhance the reliability, of the vehicle sur- 
roundings monitoring control by raising the accuracy of 
recognition of lane markers. 

[0006] To achieve these objects, a lane recognition 
apparatus for recognizing lane rharkers on a road sur- 
face based on a pair of stereo images of a scene in front 
of an own vehicle obtained from a stereoscopic camera, 
comprises a stereo image processing means fox calcu- 
lating a distance information of an image based on the 
pair of the irinages. a lane delecting means for detecting 
a lane marker of a lane in the image based on a bright^ 
ness information of the image and the distance informa- 
tion calculated by the stereo image processing means 
and a lane position recognition means for recognizing a 
position of the lane marker in real space based ori a po- 
sition of the lane marker detected by the lane detecting 
means and the distance information calculated by the 
stereo image processing means. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0007] 

Fig. 1 is a block diagram showing a stereoscopic 
type vehicle surroundings monitoring apparatus ac- 
cording to the present invention; 
Fig. 2 is a flowchart showing steps for recognizing 
lane markers according to the present invention: 
Fig. 3 is a flowchart showing steps for detecting lane 
markers according to the present invention; 
Fig. 4 is a diagram for explaining the direction of 
detecting of lane markers in an image area; 
Fig. 5 is a diagram for explaining the range of pre- 
detecting of lane markers in an image area; 
Fig. 6 is a diagram showing a detection area estab- 
lished in accordance with a mode; 
Fig. 7 is a diagram showing a brightness character- 
istic oT a horizontal line in which a lane marker ex- 
ists; 

Fig. 8 is a diagram of a lane model: 

Fig. 9 is an explanatory view of the establishment 

of a detection area in a mode 2: 

Fig. 10 is an explanatory view of the establishment 

of a detection area in a mode 0; 

Fig. 11 is a view for explaining the enlargement of 

a detection area: 

Fig. 12 is a diagram showing a difference of bright- 
ness characteristic according to a difference of ob- 
jects on a road surface; 

Fig. 13 is a diagram showing an image in which a 
preceding vehicle is projected: 
Fig. 14 is a diagram showing a relationship between 
brightness and distance in the image illustrated in 
Fig. 13: 

Fig. 15 is a diagram showing an image in which a 
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trace by braking is projected: 
Fig. 16 is a diagram showing a relationship between 
brightness and distance in the image illustrated in 
Fig. 15; 

Fig. 1 7 is an explanatory view of a method of calcu- 
lating a pitching motion of an own vehicle: ■ 
Fig. 18 is an explanatory view of a wide detection: 
Fig. 19 is a diagram for explaining a relationship be- 
tween the presence of an up-and-down and the 
configuration of a curve; and 
Fig. 20 is a diagram for explaining a weighing in ac- 
cordance with the position in the process of evalu- 
ating lane candidates. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0008] Referring now to Fig. 1. numeral 1 denotes a 
stereoscopic camera for imaging the scenery ahead of 
an own vehicle. The stereoscopic camera 1 is installed 
in the vicinity of a room mirror and is constituted by a 
pair of cameras 2a. 2b incorporating image sensors 
such as CCD and the like therein. The cameras 2a, 2b 
are transversely mounted at a specified interval of dis- 
tance (base line length). The camera 2a (right camera) 
is referred to as a main-camera for obtaining reference 
images and is mounted on the right side of the vehicle 
when viewed from a driver. On the other hand, the cam- 
era 2b (left camera) is referred to as a sub-camera for 
obtaining comparison images and is mounted on the left 
side of the vehicle when viewed from the driver. 
[0009J Analogue images outputted from the respec- 
tive cameras 2a. 2b in a synchronous timing are con- 
verted into digital images having a specified number of 
graduations (for example. 256 graduations in the gray 
scale) by AJD converters 3, 4. Further, with respect to 
thus digitalized images, an image correcting section 5 
performs a correction of brightness, a geometrical trans- 
formation of images and the like. 
[0010] Through these image processes, reference 
image data composed of 51 2 pixels horizontally and 200 
pixels vertically are generated from output signals of the 
main camera 2a and comparison image data having the 
same vertical length (200 pixels) as the reference image 
and a horizontal length larger than the reference image 
(640 pixels) are generated from output signals of the sub 
camera 2b. The reference image data and the compar- 
ison image data are stored in an image data memory 8. 
[0011] A stereo image processing section 6 calcu- 
lates a parallax based on the reference image data and 
comparison image data. Since one parallax is calculat- 
ed for every pixel block composed of 4 x 4 pixels, max- 
imum 128 x SO pieces of parallaxes are calculated for 
one frame of the reference image. With respect to a cer- 
tain pixel block (hereinafter, referred to as an object pixel 
block) in the reference image, an area having a bright- 
ness correlation with the object pixel block, that is. a cor- 
relation object is searched and identified in the compar- 



ison image. Hereinafter, this process is referred to as 
"stereo matching". As well known, the distance to an ob- 
ject projected in a stereo image is presented as a hori- 
zontal deviation amount between the reference image 

5 and comparison image. Accordingly, in searching the 
comparison image, the search may be performed along 
a horizontal line (epi-polar line) having the same j coor- 
dinate value as the object pixel block in the reference 
image . The stereo image processing section 6 evalu- 

10 ates a correlation with the object pixel block for every 
pixel block of the comparison image while shifting one 
pixel by one pixel on the epi-polar line. 
[0012] The correlation between two pixel blocks can 
be evaluated by calculating a city block distance CB ex- 

15 pressed in Formula 1. for example: 

[Formula 1] CB = |p1 ij - p2ij| 

20 where CB is a city block distance, plij is a brightness 
value of a pixel at the address ij in the reference image 
and p2ij is a brightness value of a pixel at the address 
ij in the comparison image. The city block distance CB 
is a sum of the difference (absolute value) of brightness 

25 values plij and p2ij over entire pixel blocks. The corre- 
lation between both pixel blocks becomes larger as the 
difference is small. 

[0013] The city block distance CB is calculated for 
every pixel block existing on the epipolar line . when the 

30 value of city block distance CB is smallest at a certain 
pair of pixel blocks, the other pixel in the comparison 
image is judged to be a correlation object of the object 
pixel block. The deviation amount between the object 
pixel block and the identified correlation object is a par- 

35 allax. 

[0014] Further, the stereo image processing section 
6 evaluates an existence or a non-existence of. horizon- 
tal brightness edges of the pixel block, a relationship be- 
tween a maximum and minimum values of the city block 

40 distance CB calculated on the same epi-polar line and 
the like. Further, in order to raise the reliability of paral- 
laxes, a filtering process is applied to the parallaxes 
based on the result of the evaluation. As a result of the 
filtering process, only parallaxes regarded as effective 

45 are outputted. A hardware construction for calculating 
the city block distance CB and the filtering process are 
described in detail in the Japanese Patent Application 
Laid-open No. Toku-Kai-Hei 5-114099. If necessary, the 
reference should be made to the disclosure. After those 

50 processes, the calculated effective parallax are stored 
in a distance memory 7. 

[001 5] A micro-computer 9 or functionally, a road rec- 
ognition section 10 recognizes road configurations (lane 
markers) ahead of the vehicle based on the reference 
55 image data (brightness information) stored in the image 
data memory 8 and the parallaxes stored in the distance 
memory 7. The road configurations can be identified by 
recognizing the configurations of left and right lane 
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markers (white markers, no passing lines and the like) . 
The primary feature of the embodiment lies in this lane 
recognition and more detailed descriptions of the recog- 
nition method will be made hereinafter. 
[0016] A solid object recognition section 11 recogniz- 
es solid objects (preceding vehicles and the like) in front 
of the vehicle. In recognizing road configurations or solid 
objects, vehicle behavior information from a vehicle 
speed sensor (not shown) or a steering angle sensor 
(not shown) or navigation information from a navigation 
apparatus (not shown) are taken into consideration. Fur- 
ther, a processing section 1 2 arouses a driver's attention 
by a warning device 18 such as display, speaker and 
the like or, if necessary, controls miscellaneous control 
sections 1 3 through 1 7. in case where it is judged based 
on information from the recognition sections 10 and 11 
that a warning is necessary. For example, sometimes 
an automatic transmission control section 13 is instmct- 
ed to carry out a shift-down and sometimes an engine, 
control section 17 is instructed to reduce engine power. 
Other appropriate vehicle controls may be instructed to 
an anti-lock brake system (ABS) control section 14. a 
traction control system (TCS) control section 1 5 or a ve- 
hicle behavior control section 16 for controlling torque 
distributions and rotation speeds of respective wheels. 
[0017] Fig. 2 is a flowchart showing steps for recog- 
nizing lane markers. The road recognition section 10 re- 
peatedly carries out steps shown in this flowchart until 
one picture image is completed. In this specification, the 
one picture-full data is referred to as "one frame**. 
[0018] Before entering the description of respective 
steps shown in Fig. 2. first steps for completing one 
frame of the reference image will be described by refer- 
ence to Fig. 4. 

[0019] The image in which lane markers are detected 
is referred to as "reference image" having an area size 
of 512 X 200 pixels. In descriptions hereinafter, the ref- 
erence image is referred to as just "image". The lane 
detection is performed in the horizontal direction or in 
the vertical direction for every horizontal line, as illus- 
trated in Fig. 4. A plurality of horizontal lines are estab- 
lished in the reference image from a detection starting 
line js to a detection ending line je. These starting and 
ending lines may be established fixedly or may be es- 
tablished variably according to the pitching condition of 
the own vehicle. 

[0020] The detection process in the vertical direction 
is performed from the near side to the far side, namely, 
from below to above in the image, basically for every 
horizontal line. In this embodiment, the horizontal line is 
one pixel wide, however, it may be a plurality of pixels 
wide. Further, in order to reduce the amount of calcula- 
tion, the interval of sampling, that is. the interval of the 
horizontal line may be established to gradually increase 
as the detection process goes downward of the image. 
The position of lane marker in an entire image is deter- 
mined by combining the position of partial lane markers 
on respective horizontal lines. The reason why the de- 



tection in the vertical direction is performed in such an 
order as going from below to above is that short distance 
lane markers can be detected relatively easily and in- 
clude numerous data concerning iane markers. This is 

5 advantageous in making a detection process of the next 
horizontal line based on the position of lane markers al- 
ready detected in the same frame. Because of this rea- 
son, the parallaxes obtained from the stereo image 
processing section 6 are successively processed in an 

10 order from the short distance side (below of image) to 
the long distance side (above of image) . 
[0021] Further, the detection process in the horizontal 
direction is performed successively for every one pixel 
from the center of the image to left and right side, re- 

15 spectively, that is. the detection of left lane markers is 
performed from right to left and the one of right lane 
markers is performed from left to right This is because 
the inside of left and right lane markers is a road surface 
whose brightness condition is easy to be estimated and 

20 consequently there is a small possibility of detecting 
lane markers erroneously, compared to the detection in 
an inverse direction. On the other hand, since normally 
miscellaneous objects such as gutters and the like are 
projected on the outside of the lane mariners, it is not 

25 easy to estimate the brightness condition of all objects 
likely to exist at that area. Accordingly, if the detection 
is performed from the outside to inside of lane marker, 
there is a possibility that the erroneous detection of lane 
markers often happens and it takes time to perform a 

30 detection process. 

[0022] Thus, the detection process along a horizontal 
line starts from the inside of lane marker and ends upon 
the detection of "a lane end point**. The term "lane end 
point" will be described hereinafter. 

35 [0023] Referring to a flowchart shown in Fig. 2. at a 
step 1. first the road recognition section 10 proceeds in- 
itializing processes. In this embodiment, the lane recog- 
nition is performed in a plurality of segments, respec- 
tively. The initializing processes at the step 1 include es- 

"to tablishments of a segment for ending the lane detection, 
an arrangement for indicating what segment the respec- 
tive horizontal lines belong to. a line for correcting a road 
height and the like. 

[0024] Next, at a step 2, the position of lane mari<ers 
45 is detected in an image (reference image) of one frame. 
Since the lane detection is executed for every horizontal 
line in order, the position of lane markers is a point on 
the horizontal line (hereinafter, referred to as "lane 
point**) and the lane point is detected. Fig. 3 is a flow- 
so chart showing detailed steps of lane detection process- 
es. 

[0025] An initial value of a counter] indicating the ver- 
tical position of the horizontal line is set to 0. When the 
counter value j is equal to 200 at a step 29. the routine 
55 ends. Since the horizontal line indicated by the counter 
j is an object of process, the horizontal line is referred 
to as "object line j**. 

[0026] As described above, the horizontal line on 
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which the lane detection is performed starts with a de- 
tection start line js and end with a detection end line je. 
Accordingly, with respect to horizontal lines below the 
detection start line is. the lane detection is not per- 
formed. Consequently, only the increment of the counter 
j is carried out at the step 29 until the counter j counts js. 
[0027] When the increment of the counter j goes on 
and the object line j becomes a pre horizontal line jpre. 
the program goes from a step 21 to a step 22 as a result 
of the positive judgment. The pre horizontal line jpre is 
four consecutive horizontal lines (js-4. js-3. js-2 and js- 
1) located immediately below the detection start line js. 
At a step 22, a road surface brightness Aroad is detected 
before the start of the lane detection. The road surface 
brightness Aroad is. as shown in Fig. 5. calculated 
based on brightness within a left pre-detection range 
PRE1 and a right pre-detection range PRE2. respective- 
ly. The pre-detection ranges PRE1. PRE2 are constitut- 
ed by four pre-horizontal lines jpre and a detection start 
line js and are established at an area except the central 
portion of the image. The left and right pre-detection 
ranges PRE1. PRE2 are fixedly established at an area, 
where the lane markers are likely to be projected under 
normal driving conditions. Further, a histog ram of bright- 
ness is calculated in respective detection ranges PRE1 . 
PRE2, letting a brightness value most frequently ob- 
served be a road surface brightness Aroad in the re- 
spective ranges, namely, in the area of left and right lane 
markers. The road surface brightness Aroad is used as 
a base value (base value of threshold value) in judging 
the left (or right) lane marker at a step 24. 
[0028] Since a threshold value for judging lane mark- 
ers is established for each horizontal line, it is necessary 
to identify the road surface brightness Aroad in the vi- 
cinity of the line. Since the road surface brightness 
Aroad varies according to camera angles or areas on a 
road surface, in order to raise the accuracy of lane de- 
tection, it is preferable to detect the road surface bright- 
ness Aroad independently for every object line. Specif- 
ically, the road surface brightness Aroad is calculated 
fro^n five horizontal lines (one object line and four hori- 
zontal lines immediately therebelow). Accordingly, in or- 
der to detect lane markers of the detection start line js. 
brightness information of four horizontal lines immedi- 
ately below the line js (these are not objects of lane de- 
tection) is needed. For that reason, in case where it is 
judged at the step 21 that the object line j is a pre-hori- 
zontal line jpre, at a step 22 a road surface brightness 
Aroad is determined beforehand for the preparation of 
the lane detection in the detection start line js. 
[0029] Since the object line j shifts from below to 
above, the road surface brightness Aroad of the object 
line j is calculated based on brightness information of 
respective lines already detected and stored in RAM of 
the micro-computer 9. Therefore, after the object line 
reaches a detection start line js. the step 22 is not carried 
out and thereafter, the detection of road surface bright- 
ness Aroad for respective object lines j is performed at 



a step 24. 

[0030] When the object line j reaches a detection start 
line js. the program goes from the step 21 directly to the 
step 23. At the step 23. a detection area Rj is established 

5 on the object line j. The detection area Rj specifies a 
range of lane detection on the object line j. being estab- 
lished individually for every horizontal line. Three estab>- 
lishment modes are prepared as follows in establishing 
the detection area Rj. The establishment nKide is 
changed over according to the detecting situation of 
lane marker in the previous horizontal line. By selecting 
a most appropriate establishment mode for every hori- 
zontal line, it is possible to establish an area as narrow 
as possible and capable of surely detecting lane mark- 

is ers. 

[Establishment mode of detection area] 

[0031] Mode 0: A mode in which the position of lane 
marker (lane position) on the object lirfe j is estimated 
based on the detection result of a previous horizontal 
tine in a present frame and the detection area is estab- 
lished on the basis of the estimated lane position. 
[0032] Mode 1: A mode in which a narrow detection 
area is established using the result of recognition (lane 
position) in a previous frame. 

[0033] Mode 2: A mode in which a wide detection area 
is established using the result of recognition (lane posi- 
tion) in a previous frame. 

[0034] Fig. 6 is a diagram showing a detection area R 
established according to a mode. The mode is set to 
Mode 2 in the initial condition. Accordingly, a wide de- 
tection area Rjs is established on the detection start line 
js on the basis of a lane position detected in a previous 
frame. The lane detection is proceeded for each line in 
Mode 2. From the view point of reducing processing time 
and raising the accuracy of detection, it is necessary to 
establish an area as narrow as possible. Hence, the size 
(width) of the detection area Rj of the object line j is reg- 
ulated based on "reliability of lane markers" detected in 
the previous frame. That is. as "reliability of lane mark- 
ers'* is high, there is a larger possibility that a lane mark- 
er exists in the present frame in the same position as 
the previous frame. In this case, an allowance being es- 
tablished in consideration of behaviors of the vehicle 
such as fooA^ard and backward movements, pitching 
movement and the like, a narrow detection area Rj is 
established based on a lane point P of the previous 
frame. On the other hand, in case where "reliability of 
lane markers" is low. if a narrow detection area is estab- 
lished, there is a possibility that a lane marker goes out 
of the detection area R. Therefore, in this case, it is nec- 
essary to establish a wide detection area R. 
[0035] The reliability of lane markers is calculated at 
a step 5. The detection area is established based on this 
calculated reliability of lane markers. The specific meth- 
od of evaluation of the reliability will be described in de- 
tail hereinafter. Describing roughly, the reliability of lane 
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markers are determined taking a continuity of the lane 
position (position of lane marker) between frames and 
the number of data of the lane position. 
[0036] Further, as shown in Fig. 9. the width of the 
detection area Rj on each object line j is established to 5 
be smaller as the line j lies high in the image even with 
the same degree of reliability. The lane in the image be- 
comes narrower as it is far Hence, the width of the de- 
tection area R is established to be smaller as the detec- 
tion area lies high (far) in the image. It is the same with io 
in Mode 1 . 

[0037] First, the detection area R is established in 
Mode 2 and when the lane marker is detected in a hor- 
izontal line lager than a specified number, the mode 
shifts to Mode 0. In Mode 0. a narrowest detection area is 
R among three modes is established. In this mode, the 
result of detection before the horizontal line j-1 (line irrv 
mediately below the object line) is used. Fig. 10 is a di- 
agram for explaining the establishment of the detection 
area R in Mode 0 . First, a lane point P is detected on 20 
the horizontal line j-1 and a gradient a of the lane marker 
on the image plane (i-j plane) is calculated based on the 
lane marker already detected. When a lane point P (j-1) 
is detected on a horizontal line adjacent to the object 
linej . letting a point of intersection of the object line j and 25 
a line passing through the lane point P and having a gra- 
dient a be a lane point Pj' (temporary lane point), a de- 
tection area Rj having a specified width is established 
on the basis of the temporary lane point Pj' in consider- 
ation of curves or detection errors. 
[0038] Further, under such conditions as a lane mark- 
er is broken (for example, a passing line shown in Fig. 
6). such conditions as lane markers are stained, or such 
conditions as a lane marker is curved, horizontal lines 
on which no lane marker can not detected exist. Such 35 
horizontal lines are called "non-detectable line". If such 
non-detectable lines appear consecutively after a nar- 
row detection area R is established in Mode 0. there is 
a possibility that the detection area R is established in 
a position away from actual lane markers. Hence, as 40 
shown in Fig. 11. if no lane marker can be detected in 
an object line j and the number of non-detectable lines 
exceeds a specified value, the detection area R is grad- 
ually enlarged hereinafter. W1 shown in Fig. 11 denotes 
an allowable minimum width of the detection area R 45 
which has been established when the lane point Ps are 
detected consecutively and W2 denotes an allovyable 
maximum width of the detection area R. If a specified 
number of non-detectable lines appear consecutively 
even after the detection area R reaches the allowable 50 
maximum width W2. the mode shifts to Mode 1 in which 
the detection area R is established based on the lane 
position in the previous frame. 

[0039] At a step 24. a lane nnari<er is detected within 
the detection area Rj established on the object line j. 55 
Further, a road surface brightness Aroad with respect to 
the object line j is also calculated at the step 24. If nec- 
essary, information calculated at the step 22 (pre-detec- 



tion of road surface brightness) is used. The lane detec- 
tion method according to this embodiment is character- 
ized in using distance information of the image calculat- 
ed in the stereo image processing sectran 6 besides 
brightness information (brightness value and differential 
of brightness) of the image itself. Thus, in detecting the 
lane position in the image the use of distance informa- 
tion can enhance the accuracy of detection compared 
to the method only based on brightness information. The 
lane point P (i, j) detected for each horizontal line is 
stored in a nenrwry constituting the micro-computer 9. 
[0040] Fig. 7 is diagram showing brightness charac- 
teristics of a horizontal line containing a lane marker As 
understood from this characteristic diagram, a horizon- 
tal line where a lane marker exists, has following bright- 
ness characteristics: 

there is a strong edge intensity at the boarder of 
lane marker and asphalt surface(see Fig. 7b). 
there is a high brightness portion larger than bright- 
ness of road surfaces (see Fig. 7a). 
there is a high brightness portion equivalent to the 
width (more than 10 centimeters)of an actual lane 
marker (see Fig. 7a), 

[0041] Further the horizontal line in which a lane 
marker exists, has following distance characteristics: 

distance (parallax) is calculated at the boarder 
(edge section) of a lane marker and asphalt surface, 
an approximately constant distance is calculated. 

[0042] In this embodiment, a ca ndidate of lane marker 
is extracted based on brightness information of the im- 
age (brightness data stored in the image data memory 
8). However at this stage, it is not certain that the ex- 
tracted candidate of lane marker is on a road. Here, in 
order to raise the accuracy of lane detection, it is judged 
based on distance information (parallax stored in the 
distance data memory 7) whether or not the candidate 
is on a road. 

[0043] First, the extraction of a lane candidate (here- 
inafter referred to just as "lane") will be described. 
[0044] Referring to Fig. 7a and 7b. lane points P (start 
point Ps and end point Pe) are found based on the afore- 
said brightness characteristic according to the following 
conditions. Here, the start point Ps denotes a lane point 
at an inside boarder of a lane marker and an asphalt 
surface and the end point Pe denotes a lane point at an 
outside boarderof a lane marker and an asphalt surface. 

[Conditions of start point Ps] 

[0045] 

1. The edge intensity (differential of brightness) is 
larger than a positive threshold value Eth. 

2. Brightness is larger than a threshold value AthI 
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for judging a start point brightness, 
(Conditions of end point Pe] 
[0046] 

3. The edge intensity (differential of brightness) is 
smaller than a negative threshold value -Eth. 
[Mutual conditions of start point Ps and end point 
Pe] 

4. The interval between the start point Ps and the 
end point Pe is within a specified range (equivalent 
to the width of lane marker). 

5. An average of brightness values Aave within a 
range from the start point Ps to the end point Pe is 
larger than a threshold value Ath2 forjudging bright- 
ness of lane marker (Ath1 < Ath2). 

[0047] With respect to the condition 5. since the width 
of lane marker projected on the image differs according 
to a vertical position (j coordinate) of the horizontal line, 
the aforesaid specified range is established in consid- 
eration of the width of lane marker which will be actually 
projected on the image. 

[0048] Further, the threshold values Ath1. Ath2 are 
calculated based on the road surface brightness Aroad. 
That is, the threshold values Ath1. Ath2 are calculated 
by adding constants a. P (0<a<P) to the road surface 
brightness Aroad. respectively. The calculation of the 
road surface brightness Aroad at the step 24 slightly dif- 
fers from the calculation of the road surface brightness 
at the step 22 in the method of establishing the detection 
range R. At the step 22. the road surface brightness 
Aroad is calculated from the brightness over the entire 
areas of the pre-detection ranges .PRE1. PRE2 as 
shown in Fig. 5. On the other hand, at the step 24. the 
sampling area of brightness is established in the pre- 
detection ranges PRE1 . PRE2 excepting the outside of 
the lane. The reason why the sampling area is estab- 
lished so as not to include the outside of the lane is to 
prevent an inappropriate road surface brightness Aroad 
from being calculated due to objects outside of the lane. 
In this state, since the lane point Pj is not yet identified 
on the object line j. the center point (I. j) of the detection 
area Rj is regarded as the lane point Pj. Further, in the 
same way as the step 22, a histogram of brightness of 
pixels in the sampling area is calculated, letting the 
brightness value most frequently observed be a road 
surface brightness Aroad. 

[0049] When brightness of pixels is referred to on the 
object line j in a detection order (leftward for the left lane 
marker and rightward for the right lane marker), if a point 
satisfying both of the aforesaid conditions 1. 2 is found, 
this is a start point Ps. Further, if a point satisfying the 
condition 3 is found, that is an end point Pe. Then, it is 
judged whether or not the interval between the start 
point Ps satisfies the condition 4 and the average bright- 
ness value Aave satisfies the condition 5. 



[0050] Fig. 12 is a diagram showing a difference of 
the brightness characteristic between a lane marker and 
others else(for example, stains and the like). As under- 
stood from the drawing, in case where objects other than 

5 lane markers exist on the road surface, the boarder of 
those and asphalt surface is mostly indistinct. Also, 
there is a tendency that the edge intensity of those ob- 
jects has lower peaks than that of lane markers. From 
this point of view, if an appropriate threshold value Eth 

10 is selected, it is possible to clearly discriminate lane 
markers from other objects. 

[0051] Further, in this embodiment, as shown in Fig. 
7. two threshold values Athi, Ath2 (Ath1< Ath2) are es- 
tablished in order to raise the accuracy of lane detection. 

15 As understood from the relationship between brightness 
characteristic and edge intensity as shown in Fig. 7. the 
brightness when the edge intensity forms a positive 
peak is fairly smaller than the maximum brightness val- 
ue. In order to detect the start point Ps properly, it is 

20 necessary to establish the threshold value Athi forjudg- 
ing a start point brightness to be relatively low. However, 
if the threshold value Athi is applied uniformly, that is. 
the threshold value Ath1 is used also for the threshold 
value Ath2 forjudging brightness of lane markers, there 

25 is a possibility that objects other than lane markers are 
detected as lane markers. For that reason, the threshold 
value Athi forjudging the brightness at the detection of 
edge is established separately from the threshold value 
Ath2 forjudging the brightness condition over the entire 

30 lane marker. 

[0052] A point satisfying these conditions on the ob- 
ject line j is a lane point Pj. It is not certain whether or 
not the lane point Pj exists on the road. Hence, in order 
to raise the accuracy of the result of detection, it is ver- 

35 ified whether or not the lane point Pj obtained from 
brightness information exists on the road. For example, 
a picture image as shown in Fig. 13 being considered, 
the picture image shows a brightness characteristic as 
shown in Fig. 14a depending upon colors, structures, 

40 direction of a vehicle traveling ahead of an own vehicle. 
In this case, since the detection is dependent only upon 
brightness information, there is fear that a pillar on the 
preceding vehicle is detected as a lane marker. In this 
case, if distance information is concurrently used, as 

45 shown in Fig. 14b, since there is a clear difference be- 
tween a distance to the pillar of the preceding vehicle 
and that to the lane marker on the road, the verification 
by distance information enables to detect only lane 
marker. Respective lane points P (i. j) thus obtained from 

50 both brightness and distance information are stored in 
RAM of the micro-computer 9. 

[0053] According to the lane detection method de- 
scribed above, first a lane marker is detected form 
brightness information and then the result of detection 
55 is verified based on distance information. Alternatively, 
first objects on the road are extracted by processing dis- 
tance information and then a lane marker may be iden- 
tified based on brightness information. For example, a 
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picture image as illustrated in Fig. 15 being considered, 
in case such a picture image, distance (parallax) to por- 
tions such as a brake trace and a Lane marker is calcu- 
lated. Therefore, even when there are objects other than 
the road surface, those objects can be deleted by use 
of distance information. Further, when brightness infor- 
nnation is processed in the vicinity of the position where 
the distance is detected, only the lane marker can be 
extracted by judging the difference of brightness char- 
acteristic between lane markers and brake traces as 
shown in Fig 16a. 

[0054] In case where the object line j is a correction 
line jamd. the program goes from the step 25 to a step 
26. The correction line jamd denotes a group of horizon- 
tal lines from ja to jb (jb ^ja) established beforehand in 
a relatively low part of the image. At the step 26. a pitch- 
ing amount of the own vehicle is calculated based on 
respective lane points P detected on respective correc- 
tion lines jamd and a parallax d at a lane point R The 
parallax d is a parallax calculated for pixel blocks includ- 
ing the lane points P and is stored in the distance data 
menrrary 7. 

[0055] The detection of the pitching condition is inr>- 
portant in raising the accuracy of recognition not only of 
a road (lane markers) but also of an object (preceding 
vehicle) . In this embodiment, in order to discrirhinate a 
preceding vehicle from an object on the road, for exam- 
ple a small sink of the road surface, paint and the like, 
a road height model is used. That is. in case where an 
object exists on a higher side than the road surface 
specified by the road height model, that object can be 
regarded as a preceding vehicle. 

[0056] However, when the recognition process of sol- 
id objects is started using the road height model calcu- 
lated this time, actually there are restrictions such as 
processing time, hardware ability and the like. Accord- 
ingly, these processes can not proceeded in parallel and 
in the present recognition of solid objects, a road height 
model calculatedpreviously is used. Hence, when a 
pitching motion of the own vehicle occurs, the height of 
the'road seen from the stereoscopic camera 1 changes 
and as a result a deviation is caused between the height 
of the road of the camera and the height of the road cal- 
culated by the road height nrtodel. Due to this deviation, 
there is a possibility that a solid object is erroneously 
recognized. For that reason, at the step 26. the pitching 
condition of the own vehicle is detected before calculat- 
ing the lane model at the step 3. 

[0057] Specifically, the amount of pitching is calculat- 
ed as follows. First, at a step 24, respective lane points 
P(i. j) in the horizontal line group Jamd (correction line) 
are detected. The respective points P(i, j) indicate the 
position of a lane marker on a relatively lower side of the 
image, that is, a lane marker located in a sort distance 
(distance up to around 20 meters) . The road height in 
the present frame iscalculated based on the lane points 
P thus sampled. 

[0058] Next, the parallax d with respect to the respec- 



tive lane points P is calculated. The parallax d of the 
respective lane points P is a parallax calculated with re- 
spect to a pixel block including the lane point P and 
stored in the distance data memory 7. Thus, coordinates 
^ (K j) of the respective lane points P and a parallax d are 
identified as a set of coordinates (i. j, d). A set of coor- 
dinates (X. Y. Z) of a lane point P in real space can be 
unconditionally identified from the set of coordinates (i. 
j. d) thus obtained. X denotes a position in a widthwise 

10 direction of the vehicle. Y denotes a position in a vertical 
(height) direction of the vehicle and Z denotes a position 
in a longitudinal direction of the vehicle, that is. a dis- 
tance. A specific way of coordinate transformation from 
(i, j. d) to (X. Y. Z) will be described at a step 3. 

15 [0059] Thus obtained lane points P are plotted on a 
Y-Z (Y: height, Z: distance) plane and an approximation 
line L' is calculated by applying the least square method 
to the plotted lane points P. Further, a gradient a of the 
approximation line L' is calculated with respect to the 

20 road height model (road height L in a specified distance 
from the own vehicle) calculated in the previous frame. 
The pitching amount can be estimated from the gradient 
a. If the gradients a is positive, it means that the own 
vehicle is directed downward. As the absolute value of 

25 the gradient a is large, the pitching amount increases. 
[0060] The road height model calculated in the previ- 
ous frame is corrected by thus calculated gradient a. 
The corrected road height model is used for calculating 
a road surface in the solid object recognition. Since the 

30 effect of pitching is reduced when the present solid ob- 
ject recognition is performed based on the corrected 
road height nrradel. the accuracy of recognition can be 
raised. 

[0061] Further, a roll angle of the own vehicle or a 

35 bank angle of a road can be obtained by calculating the 
height of left and right lane markers separately. These 
information is effective not only to raise the accuracy of 
solid object recognition but also to investigate for erro- 
neous recognition. 

to [0062] In case where the object line j is a wide detec- 
tion line jwide. the program goes from a step 27 to a step 
28. The wide detection line jwide is a particular horizon- 
tal line established between the detection start line js 
and the detection end line je. At the step 28. a wide de- 

45 tection. that is. a lane detection over the entire area of 
the wide detection line jwide is performed. 
[0063] In the lane detection at the step 24. since the 
detection area R is established on the object line j for 
the reason of the detection accuracy and processing 

50 time, the lane detection is not performed outside of the 
detection area R. Accordingly, if the lane marker detect- 
ed in a given horizontal line is right, a right lane marker 
is mostly detected in subsequent line. However, in case 
where a plurality of lane markers exist on one side of 

55 the road as shown in Fig. 18, there is a likelihood that a 
first lane marker is erroneously detected. For example, 
wherein a lane marker SL should have been delected, 
a lane marker SL* outside thereof is detected. In this 
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case, there is a strong possibility that the same lane 
marker SL' is successively detected also in subsequent 
horizontal lines. Taking such situations into considera- 
tion, a wide detection is a process in which all candi- 
dates of lane markers SL. SL* on the wide detection line 5 
jwide are detected. In case where a plurality of lane can- 
didates SL. SL'are detected at a step 28. flag informa- 
tion showing a presence of lane candidates and respec- 
tive lane points PsI. Ps1' are stored in RAM of the micro- 
computer 9. These information is used for evaluation of 
lane candidates at a later step 4. 
[0064] At a step 29. a counter j is incremented, the 
object line shifting from a horizontal line j to a horizontal 
line j+1 . Further, according to the judgment at a step 30. 
aforegoing steps 21 through 29 are carried out for every '5 
horizontal line. When a process of a horizontal line lo- 
cated in the uppermost position, that is, a process of a 
horizontal line j=199 finishes, this routine ends. 
[0065] After the lane detection process at the step 2 
finishes, at a step 3 "lane model" showing a three-di- 20 
mensional (real space) position of lane hiarker is calcu- 
lated . For that purpose, "lane points P" are read out from 
RAM in the micro-computer 9 and at the same time par- 
allax data stored in the distance data memory 7 are read 
out. ^5 
[0066] The lane model is calculated as follows. First, 
a parallax d of a lane point P is identified from parallaxes 
of one frame. Further, coordinates (i. j. d) for all lane 
points P are obtained from respective lane points P (i. 
j) and parallaxes d. 

[0067] A position (X, Y. Z) of left and right lane markers 
in real space is determined unconditionally according to 
the following formula: 



[0068] 

[Formula 2] Z = KZH/(d - DP) 
X = r/2 + Z PW (i - IV) 
Y = CH - Z PW (j - JV) 

45 

where KZH is a base length of camera/angle of horizon- 
tal visual field. DP is a parallax of a vanishing point. CH 
is installation height of stereoscopic camera, r is instal- 
lation interval of stereoscopic camera. PW is an angle 
of view per one pixel. IV is i-coordinate of an infinite point so 
on image in front of vehicle and JV is j-coordinate of the 
infinite point. 

[0069} . The coordinate system of image comprises an 
origin located at the left below corner, an i-axis extend- 
ing in the horizontal direction and a j-axis extending in 55 
the vertical direction. Unit of coordinate axes is one pix- 
el. On the other hand, the coordinate system of real 
space which is established on the basis of the position 



of an own vehicle, comprises an origin located on the 
road surface right undemeath the center of the stereo- 
scopic camera 1. a X-axis extending in the widthwise 
direction of the vehicle (defining right direction as posi- 
tive), a Y-axis extending in the vertical direction of the 
vehicle (defining upward direction as positive) and a Z- 
axis extending in the longitudinal direction of the vehicle 
(defining forward direction as positive). Therefore, if a 
road on which the own vehicle is traveling is even, a X- 
Z plane (Y = 0) agrees with a road surface of the road. 
[0070] A lane model is calculated from thus calculated 
position (X. Y. Z) of the lane point P in real space. The 
lane model expresses the left and right lanes within a 
recognition range (distance from the camera position to 
a specified position)as a combination of plurality of bro- 
ken lines each of which is a three-dimensional approx- 
imation line calculated per specified interval (segment) 
respectively. An example of the lane model is shown in 
Fig. 8. 

The recognition range is divided into seven segments, 
letting a distance to the end point of respective intervals 
be Z1. Z2. Z3, Z4. Z5. Z6 and Z7 in a nearest order. 
Then, an approxinnation line is prepared by applying the 
least square method to lane points P within respective 
segments. Further, parameters a. b. c and d of approx- 
imation lines of segments are calculated with respect to 
left and right lane markers, respectively The road sur- 
face is an area enclosed by left and right approximation 
lines L. R. This lane model also has a linear equation 
extending horizontally (X direction) or a horizontal con- 
figuration model and a linear equation extending verti- 
cally (Y direction) or a road height model, expressing 
road configurations such as gradients, unevenness of 
roads and the like. 

[Formula 3] 
(Horizontal configuration model) 
Left lane marker: 

Right lane marker: 
X = ap Z + bR 

(Road height model) 
Left lane marker: 

Y = CLZ + dL 
Right lane mariner: 

Y = Cr Z + dR 
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[0071] In the aforesaid example, a value of X (width- 
wise position of lane marker) is calculated based on the 
coordinate transformation formula shown in Formula 2. 
Otherwise, the widthwise position of lane marker can be 
calculated by projecting the lane point P(i. j) on the road 
surface (an area enclosed by left and right lane markers 
in the road height model). Thus, a lane model showing 
a three-dimensional configuration is calculated. 
[0072J Since the unevenness condition of a road sur- 
face can not be judged according to a method of lane 
recognition using only brightness information, an accu- 
rate lane configuration in real space can not recognized. 
For example, as shown in Fig. 19. comparing two roads 
having the same curvature of radius, one road which is 
even differs from the other one which is uneven in the 
condition of images. As understood from the drawing, 
in case where there is an unevenness (in this case, the 
other road changes from even to upgrade on the way) . 
the curvature of the road seems to have a small curva- 
ture (large radius) . This has an adverse effect on the 
control of a warning device, automatic brake system and 
the like. On the other hand, according to the embodi- 
ment of the present invention, since both brightness and 
distance information is used, a three-dimensional posi- 
tion information concerning lane markers can be ob- 
tained. Thus obtained positional information enables to 
discriminate objects on the road (paint, manholes, 
patches and the like) from other obstacles else (man. 
vehicle, walls, buildings and the like) with accuracy 
[0073] At a step 4. the evaluation of candidates for 
lane marker is performed based on the result of the wide 
detection at a step 28. That is. in case where the object 
line is a wide detection line jwide and a plurality of ob- 
jects supposed to be candidates for lane markers exist, 
respective candidates are evaluated. A time-versus his- 
togram (accumulation of position data of lane markers 
obtained for each frame) is obtained with respect to the 
position of the candidates for lane markers detected (po- 
sition on the horizontal line). At this time, the histogram 
is calculated being weighted by the position in which the 
candidates for lane markers are detected, as shown in 
Fig. 20. 

[Evaluation by position on horizontal line] 
Establishment of weight 

[0074] As shown in Fig. 20. weight is established ac- 
cording to the position on the horizontal line so as to 
become largest in a specified area on the left and right 
sides with respect to the center of the vehicle. This area 
corresponds to an area in which lane markers are in- 
cluded with high likelihood. Accordingly, there is a strong 
possibility that the candidates existing in that area are 
real lane markers. 



Preparation of histogram 

[0075] The horizontal line being divided into a plurality 
of areas having a predetermined interval, the number of 
observations versus time is counted. In an area includ- 
ing lane markers, the value of the histogram corre- . 
spending to the area increases. In order to prevent the 
value from excessively increasing, an upper limit is pro- 
vided. Further, in the other area including no lane mark- 
ers, the value of the histogram corresponding to the oth- 
er area decreases. The lower limit is 0. 
[0076] In addition to the aforesaid method, weight 
may be varied according to vehicle speeds as follows. 



Establishment of weight 

[0077] Weight is established to be small at low 
speeds. At low speeds, objects other than lane marker 
(paint on the road surface and the like) are detected suc- 
cessively and the histogram is accumulated due to the 
objects. To reduce the increasing rate of histogram, a 
small weight is applied at low speeds. Further, in case 
where speeds are smaller than a certain value, the de- 
tection may be stopped. Further, weight is established 
to be large at high speeds. Specifically, weight is estab- 
lished so as to be able to detect a dotted line (passing 
lane marker) with 6 meters interval. 
[0078] The candidates for lane markers are evaluated 
based on a histogram prepared according to the afore- 
said method. A most probable candidate is referred to 
as a lane marker. Hereinafter, thus detected lane marker 
is corrected at a step 7. 

[0079] At a step 5. the reliability of the lane marker 
(lane model) is calculated. As described above, the po- 
sition of the lane marker in real space is expressed in 
the lane model. Since the lane model is used also for 
other purposes such as recognition of solid objects, mis- 
cellaneous controls (warning control) and the like, it is 
necessary that an index representing reliability, that is, 
a degree of reliability is expressed in figures. The lane 
model is not corrected by this degree of reliability. The 
degree of reliability is for giving an index of reliability to 
other processes for recognition using the lane model. 
The degree of reliability calculated this time is used for 
determining the size of the detection area R to be es- 
tablished next at the step 23. 

[0080] The degree of reliability is calculated in consid- 
eration of the number of the lane points P detected and 
the continuity of the lane points P detected in the previ- 
ous frame. That is. in case where the lane points P are 
detected successively, it is judged that the. reliability is 
high. Then, the utility of the reliability can be enhanced 
further more by calculating the reliability of lane marker 
for every segment as shown in Fig. 8. 
[0081] The reliability is calculated according to the fol- 
lowing method for example. First, the degree of reliabil- 
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ity D is calculated for each segment according to the 
presence or absence of data. The degree of reliability D 
calculated in the previous cycle for a given segment is 
incremented or decremented according to the number 
of sample points as the calculation base of the lane mod- 5 
el for that segment. Thus, the present degree of reliabil- 
ity D is calculated. Accordingly, if the sample points are 
calculated successively within a certain segment, the 
degree of reliability D increases. To prevent an exces- 
sive increase, an upper limit is provided. On the other io 
hand, in case where no sample points are detected, the 
degree of reliability D is decremented. 
[0082] Further, the positional deviation between the 
previous lane model and the present lane model is ob- 
tained for every segment and the degree of reliability D 15 
is incremented or decremented according to the devia- 
tion amount. Accordingly, in case where the positional 
deviation occurs successively, the reliability goes down. 
[0083] Further, at a step 6. the parallelization of left 
and right lane markers is evaluated. Normally, left and 20 
right lane markers are parallel wit each other. Therefore, 
otherwise, there is a strong likelihood that the lane mak- 
ers are recognized erroneously. At the step 6, the par- 
allelization of the left and right lane model is evaluated. 
Further, in case where some abnormality is found with 25 
respect to the parallelization. it is judged that those lane 
markers have been recognized erroneously and at a 
step 7 the present lane model is corrected. For example, 
in case where the interval of the left and right lane mark- 
ers becomes larger (wider) as they goes far, either one 30 
is treated as abnormal. Such a case often happens 
when the lane is pronged. In this case, it is judged that 
a lane marker extending in the same direction as the 
vehicle traveling direction is a right one. correcting the 
abnormal one so as to run in parallel with the other one 35 
(step 7) . Inversely, in case where the interval of the left 
and right lane markers becomes smaller (narrower) as 
they goes far. a lane marker with lower reliability is treat- 
ed as erroneous recognition, the lane model being cor- 
rected (step 7). 

[0084] Thus, the processes in the present frame finish 
through a series of steps (from step 1 to step 7). 
[0085] As understood from descriptions mentioned 
above, according to the embodiment, not only bright- 
ness information of images but also distance informa- 45 
tion obtained from the stereo image processing is used 
for the recognition of lane markers. In case of the prior 
art employing a monocular camera, distance informa- 
tion (three-dimensional information) of lane markers can 
not be obtained. Hence, there is a possibility that objects 50 
(pillars of preceding vehicle, guard rails and the like) 
whose configuration resembles that of lane markers are 
recognized as real lane makers . although those objects 
are not on the road. On the other hand, in case of em- 
ploying only distance information obtained from a ster- 55 
eoscopic camera, there is a possibility that objects (slip 
traces of tire, shadow, patches on the road surface and 
the like) whose configuration resembles that of lane 



markers but whose brightness differs from thai of lane 
markers, are recognized as real lane markers because 
objects are located on the road. According to the present 
invention, since the recognition of lane markers is per- 
formed using both of distance and brightness informa- 
tion (including brightness data and brightness differen- 
tial data), it is possible to obtain accurate lane informa- 
tion such as information with respect to the position of 
lane marker and height of lane marker Accurate lane 
information ensures a high reliability in performing a sur- 
roundings monitoring control of a vehicle. 
[0086] While the presently preferred embodiment of 
the present invention has been shown and described . 
it is to be understood that this disclosure is for the pur- 
pose of illustration and that various changes and modi- 
fications may be made without departing frohn the scope 
of the invention as set forth in the appended claims. 



Cfaims 

1. A lane recognition apparatus having an apparatus 
for recognizing lane markers on a road surface 
based on a pair of stereo images of a scene in front 
of an own vehicle obtained from a stereoscopic 
camera, comprising: 

a stereo image processing means for calculat- 
ing a distance information of an image based 
on said pair of said images; 
a lane detecting means for detecting a lane 
marker of a lane in said image based on a 
brightness information of said image: and 
a lane position recognition means for recogniz- 
ing a position of said lane marker in real space 
based on a position of said lane marker detect- 
ed by said lane detecting means and said dis- 
tance information calculated by said stereo im- 
age processing means. 

2. A lane recognition apparatus for recognizing lane 
markers on a road surface based on a pair of stereo 
images of a scene in front of an own vehicle ob- 
tained from a stereoscopic camera, comprising: 

a stereo image processing means for calculat- 
ing a distance information of an image based 
on said pair of said images; 
a lane detecting means for detecting a lane 
marker of a lane in said image based on a 
brightness information of said image and said 
distance information calculated by said stereo 
image processing means; and 
a lane position recognition means for recogniz- 
ing a position of said lane marker in real space 
based on a position of said lane marker detect- 
ed by said lane detecting means and said dis- 
tance information calculated by said stereo im- 
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age processing means. 

3. The lane recognition apparatus according to claim 

1 . further comprising: 

a detection area establishing means for es- 
tablishing a detection area for every horizontal line 
ir> said image. 

4. The lane recognition apparatus according to claim 

2. further comprising: 

a detection area establishing means for es- 
tablishing a detection area for every horizontal line 
in said image. 

5. The lane recognition apparatus according to claim 

3 wherein 

said detection area establishing means has a 
first mode in which said detection area is estab- 
lished based on the result of detection of said lane 
marker in a previous image on a horizontal line as 
a detection object and a second mode in which said 
detection area is established based on the result of 
detection of said lane nr^rker in a present image on 
a horizontal line as a detection object and said de- 
tection area is established on said horizontal line 
according to a mode selected from said first and 
second modes based on a detecting condition of 
said lane marker. 

6. The lane recognition apparatus according to claim 

4 wherein 

said detection area establishing nrieans has a 
first mode in which said detection area is estab- 
lished based on the result of detection of said lane 
marker in a previous image on a horizontal line as 
a detection object and a second mode in which said 
detection area is established based on the result of 
detection of said lane marker in a present image on 
a horizontal line as a detection object and said de- 
tection area is established on said horizontal line 
according to a mode selected from said first and 
second modes based on a detecting condition of 
said lane marker 

7. The lane recognition apparatus according to claim 

3. wherein 

said lane detecting means detects said lane 
marker within said detection area established by 
said detection area establishing means. 

8. The lane recognition apparatus according to claim 

4. wherein 

said lane detecting means detects said lane 
marker within said detection area established by 
said detection area establishing nrteans. 

9. The lane recognition apparatus according to claim 
3. further comprising: 



an evaluation means for calculating a reliabil- 
ity of said lane marker detected in a previous image. 

10. The lane recognition apparatus according to claim 
5 9. wherein 

said detection area establishing means 
changes a size of said detection area according to 
said reliability calculated by said evaluation means. 

10 11. The lane recognition apparatus according to claim 
4. further comprising: 

an evaluation means for calculating a reliabil- 
ity of said lane maricer detected in a previous image. 

15 12. The lane recognition apparatus according to claim 
11. wherein 

said detection area establishing means 
changes a size of said detection area according to 
said reliability calculated by said evaluation means. 

20 

13. The lane recognition apparatus according to claim 
9. wherein 

said evaluation means evaluates said reliabil- 
ity of said lane marker based on the number of de- 
25 tected data and a continuity between a position of 
lane marker in a present image and a position of 
lane marker in a previous image. 

14. The lane recognition apparatus according to claim 
30 11. wherein 

said evaluation means evaluates said reliabil- 
ity of said lane marker based on the number of de- 
tected data and a continuity between a position of 
lane marker in a present image and a position of 
35 lane marker in a previous image. 

15. The lane recognition apparatus according to claim 
11 . wherein 

said evaluation means divides said lane 
40 marker into a plurality of segments and evaluates 
said reliability of said lane marker in respective seg- 
ments. 

16. The lane recognition apparatus according to claim 
45 11. wherein 

said evaluation means divides said lane 
marker into a plurality of segments and evaluates 
said reliability of said lane marker in respective seg- 
ments. 

50 

17. The lane recognition apparatus according to claim 
1 . further comprising: 

a correcting means for evaluating a paralleli- 
zation of left and right lane mari<ers recognized by 
55 said lane position recognition means and for cor- 
recting said recognized position of said lane mark- 
ers based on the result of said evaluated paralleli- 
zation. 
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18. The lane recognition apparatus according to claim 
2. further comprising: 

a correcting means for evaluating a paralleli- 
zation of left and right lane markers recognized by 
said lane position recognition means and for cor- 
recting said recognized position of said lane mark- 
ers based on the result of said evaluated paralleli- 
zation. 

19. The lane recognition apparatus according to claim 

1, wherein 

said lane detecting means detects all candi- 
dates of lane markers existing on at least one hor- 
izontal line in said image by detecting said lane 
marker in all areas of said horizontal line. 

20. The lane recognition apparatus according to claim 

2, wherein 

said lane detecting rrieans detects all candi- 
dates of lane markers existing on at least one hor- 
izontal line in said image by delecting said lane 
marker in all areas of said horizontal line. 

21. The lane recognition apparatus according to claim 

1 . wherein 

said lane detecting means extracts said lane 
marker by applying a weight according to a detect- 
ing position on at least one horizontal line. 

22. The lane recognition apparatus according to claim 

2. wherein 

said lane detecting means extracts said lane 
marker by applying a weight according to a detect- 
ing position on at least one horizontal line. 

23. The lane recognition apparatus according to claim 

21. wherein 

said weight is changed according to a vehicle 
speed. 

24. The lane recognition apparatus according to claim 

22. wherein 

said weight is changed according to a vehicle 
speed. 

25. The lane recognition apparatus according to claim 

3. wherein 

said detection area establishing means 
makes a horizontal width of said detection area nar- 
rower as said detection area goes upward of said 
image. 

26. The lane recognition apparatus according to claim 

4. wherein 

said detection area establishing means 
makes a horizontal width of said detection area nar- 
rower as said detection area goes upward of said 
image. 



27. The lane recognition apparatus according to claim 

3. wherein 

said detection area establishing means calcu- 
lates a gradient of said lane marker in an image 
5 plane based on said lane marker already delected 

and based on said calculated gradient, establishes 
said detection area in a horizontal line as a detec- 
tion object. 

10 28. The lane recognition apparatus according to claim 

4. wherein 

said detection area establishing means calcu- 
lates a gradient of said lane marker in an image 
plane based on said lane marker already detected 
15 and based on said calculated gradient, establishes 
said detection area in a horizontal line as a detec- 
tion object. 

29. The lane recognition apparatus according to claim 
20 3. wherein 

said detection area establishing means, in 
case where no lane marker can not be detected in 
a horizontal line as a detection object, broadens 
said detection area in a horizontal line as a detec- 
25 tion object. 

30. The lane recognition apparatus according to claim 
4. wherein 

said detection area establishing means, in 
30 case where no lane marker can not be detected in 

a horizontal line as a detection object, broadens 
said detection area in a horizontal line as a detec- 
tion object. 

35 31. The lane recognition apparatus according to claim 
2, wherein 

said lane detecting means selects candidates 
of lane markers on a road surface based on said 
distance information calculated by said stereo im- 
-io age processing means and identifies a real lane 
marker from among said candidates of lane mark- 
ers based on said brightness information of said im- 
age, 

45 32. The lane recognition apparatus according to claim 
2. wherein 

said lane detecting.means selects candidates 
of lane markers based on said brightness informa- 
tion of said image and identifies a real lane marker 
50 from among said candidates of lane markers based 

on said distance information calculated by said ster- 
eo image processing means. 

33. The lane recognition apparatus according to claim 
55 7. wherein 

said lane detecting means judges a position 
where a horizontal edge intensity is larger than a 
positive threshold value and a position where said 



13 



25 



EP 1 089 231 A2 



26 



horizontal edge intensity is smaller than a negative 
threshold value to be a boarder of road surface and 
lane marker, respectively. 

34. The lane recognition apparatus according to claim 
8. vi^herein 

said lane detecting means judges a position 
where a horizontal edge intensity is larger than a 
positive threshold value and a position where said 
horizontal edge intensity is smaller than a negative 
threshold value to be a boarder of road surface and 
lane marker, respectively. 

35. The lane recognition apparatus according to claim 
7» wherein 

said lane detecting means judges an area 
having a larger brightness than a brightness thresh- 
old value calculated based on a road surface bright- 
ness which is calculated from a brightness of a plu- 
rality of horizontal lines located in the vicinity of a 
horizontal line as a detection object. 

36. The lane recognition apparatus according to claim 
8. wherein 

said lane detecting means judges an area 
having a larger brightness than a brightness thresh- 
old value calculated based on a road surface bright- 
ness which is calculated from a brightness of a plu- 
rality of horizontal lines located in the vicinity of a 
horizontal line as a detection object. 

37. The lane recognition apparatus according to claim 

7, wherein 

said lane detecting means judges a position 
of a boarder of road surface and lane marker based 
on a first brightness threshold value and a second 
brightness threshold value larger than said first 
brightness threshold value, said first brightness 
threshold value is a threshold value for judging a 
boarder of road surface and lane marker and said 
•second brightness threshold value is a threshold 
value for judging a brightness condition having a 
larger brightness than a road surface in a searching 
area. 

38. The lane recognition apparatus according to claim 

8. wherein 

said lane detecting means judges a position 
of a boarder of road surface and lane marker based 
on a first brightness threshold value and a second 
brightness threshold value larger than said first 
brightness threshold value, said first brightness 
threshold value is a threshold value for judging a 
boarder of road surface and lane marker and said 
second brightness threshold value is a threshold 
value for judging a brightness condition having a 
larger brightness than a road surface in a searching 
area. 



39. The lane recognition apparatus according to claim 

17. further comprising: 

a road height calculating means for calculat- 
ing a height of road for every segment of road and 

5 a road height estimating means for estinnating a 

height of road of all segments in a present image 
by correcting a height of road calculated in a previ- 
ous image of all segments based on a deviation be- 
tween a present height of road calculated in a spec- 

10 ified segment on the near skJe of an own vehicle 
and a height of road calculated in a previous image. 

40. The lane recognition apparatus according to claim 

18. further comprising: 

15 a road height calculating means for calculat- 

ing a height of road for every segment of road and 
a road height estimating means for estimating a 
height of road of all segments in a present image 
by correcting a height of road calculated in a previ- 
ous image of all segments based on a deviation be- 
tween a present height of road calculated in a spec- 
ified segment on the near side of an own vehicle 
and a height of road calculated in a previous image. 

41. A solid object recognition apparatus having an ap- 
paratus for recognizing a solid object on a road sur- 
face based on a pair of stereo images with respect 
to an image obtained by imaging a scene in front of 
an own vehicle by a stereoscopic camera, compris- 
ing: 

a stereo image processing means for calculat- 
ing a distance information with respect to said 
image based on said pair of said stereo images; 
a lane detecting means for detecting a lane 
marker in said image based on a brightness in- 
formation of said image; 
a road height calculating means for calculating 
a height of said lane marker in real space based 
on a position of said lane marker detected by 
said lane detecting means and a distance infor- 
mation calculated by said stereo image 
processing means; and 

a solid object judging means for judging an ob- 
ject projected in said image to be an solid object 
when said object exists above a road surface 
recognized by said road height calculating 
means. 

42. A vehicle behavior estimating apparatus having an 
apparatus for detecting a pitching of an own vehicle 
based on a pair of stereo images with respect to an 
image obtained by imaging a scene in front of an 
own vehicle by a stereoscopic camera, comprising: 

a stereo image processing means for calculat- 
ing a distance information with respect to said 
image based on said pair of said stereo images: 
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a lane delecting means for detecting a lane 
marker in said image based on a brightness in- 
formation of said image: 
a road height calculating means for calculating 
a height of road in real space based on a posi- 5 
tion of said lane marker detected by said lane 
detecting means and a distance information 
calculated by said stereo image processing 
means: and 

a pitching condition estimating means for esti- to 
mating a pitching condition of an own vehicle 
based on a deviation between a height of road 
calculated by said road height calculating 
means and a height of road calculated in a pre- 
vious image. 

43. The vehicle behavior estimating apparatus accord- 
ing to claim 42. wherein 

said road height calculating means calculates 
said height of road for every successive segment 20 
and estimates said pitching condition based on a 
deviation between a height of road calculated in a 
specified segment on a near side of an own vehicle 
and a height of road calculated in a previous image. 

25 

44. A vehicle behavior estimating apparatus having an 
apparatus for estimating a bank angle of a road sur- 
face based on a pair of stereo images with respect 
to an image obtained by imaging a scene in front of 

an own vehicle by a stereoscopic camera, compris- 30 
ing: 



a stereo image processing means for calculat- 
ing a distance information with respect to said 
image based on said pair of said stereo images; 
a lane detecting means for detecting a lane 
marker in said image based on a brightness in- 
formation of said image: 
a lane height calculating means for calculating 
a height of a left lane marker in real space and 
a height of a right lane marker in real space sep- 
arately based on a position of said lane marker 
detected by said lane detecting means and said 
distance information calculated by said stereo 
image processing means: and 
a bank angle estimating means for estimating 
a bank angle of a road surface based on a de- 
viation between said height of said left lane 
marker and said height of said right lane marker 
calculated by said lane height calculating 
means. 



a stereo image processing means for calculat- 
ing a distance information with respect to said 
image based on said pair of said stereo images: 35 
a lane detecting means for detecting a lane 
marker in said image based on a brightness in- 
formation of said image: 
a lane height calculating means for calculating 
a height of a left lane marker in real space and -fo 
a height of a right lane marker in real space sep- 
arately based on a position of said lane marker 
detected by said lane detecting means and said 
distance information calculated by said stereo 
image processing means: and 
a roll angle estimating means for estimating a 
roll angle of an own vehicle based on a devia- 
tion between said height of said left lane marker 
and said height of said right lane marker calcu- 
lated by said lane height calculating means. 50 

45. A road situation estimating apparatus having an ap- 
paratus for estimating a bank angle of a road sur- 
face based on a pair of stereo images with respect 
to an image obtained by imaging a scene in front of 55 
an own vehicle by a stereoscopic camera, compris- 
ing: 
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